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Abstract 

Background: The Spinal Instability Neoplastic Score (SINS) categorizes tumor related spinal instability. It has the 
potential to streamline the referral of patients with established or potential spinal instability to a spine surgeon. This 
study aims to define the inter- and intra-observer reliability and validity of SINS among radiation oncologists. 

Methods: Thirty-three radiation oncologists, across ten international sites, rated 30 neoplastic spinal disease cases. 
For each case, the total SINS (0-18 points), three clinical categories (stable: 0-6 points, potentially unstable: 7-12 
points, and unstable: 13-18 points), and a binary scale ('stable': 0-6 points and 'current or possible instability'; surgical 
consultation recommended: 7-18 points) were recorded. Evaluation was repeated 6-8 weeks later. Inter-observer 
agreement and intra-observer reproducibility were calculated by means of the kappa statistic and translated into 
levels of agreement (slight, fair, moderate, substantial, and excellent). Validity was determined by comparing the 
ratings against a spinal surgeon's consensus standard. 

Results: Radiation oncologists demonstrated substantial (k = 0.76) inter-observer and excellent (k = 0.80) intra-observer 
reliability when using the SINS binary scale ('stable' versus 'current or possible instability'). Validity of the binary scale 
was also excellent (k = 0.85) compared with the gold standard. None of the unstable cases was rated as stable by the 
radiation oncologists ensuring all were appropriately recommended for surgical consultation. 

Conclusions: Among radiation oncologists SINS is a highly reliable, reproducible, and valid assessment tool to address 
a key question in tumor related spinal disease: Is the spine 'stable' or is there 'current or possible instability' that 
warrants surgical assessment? 
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Background 

Spinal column metastases are present in up to 70% of 
cancer patients and as life expectancy improves with 
advances in oncology treatment, rates are expected to 
increase [1-3]. The modalities available to treat patients 
with metastatic spine disease continue to evolve and are 
offered by a range of healthcare professionals including 
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radiation and medical oncologists, radiologists, pain spe- 
cialists, and surgeons. The aim of these treatments is 
to optimize a patient s quality of life by providing effect- 
ive pain relief and preserving or restoring neurological 
function. 

Consultation with spine surgeons is typically initiated 
when neurological decompression or stabilization of 
impending or existing spinal instability is considered 
necessary. The Spinal Oncology Study Group (SOSG), 
consisting of 30 international spine oncology surgeons, 
recently defined spinal instability as "... loss of spinal 
integrity as a result of a neoplastic process that is as- 
sociated with movement-related pain, symptomatic or 
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progressive deformity, and/or neural compromise under 
physiological loads" [4]. Although vitally important, the 
assessment of tumor related spinal instability for the 
spine surgeon is challenging despite it being part of their 
training and practice. Being outside the surgical realm, 
instability recognition for the oncologist and other non- 
surgical members of the multidisciplinary care team is 
probably even more demanding; possibly resulting in 
under-recognition and under-referral of patients who 
may benefit from surgical intervention [5,6]. 

Recently, the SOSG developed the Spinal Instability 
Neoplastic Score (SINS), a standardized framework to 
help physicians assess and categorize spinal instability 
[7]. The purpose of SINS is to provide a tool to facilitate 
referrals for assessment and possible treatment of in- 
stability, a condition that can lead to catastrophic neuro- 
logical, deformity and painful situations. The SINS 
evaluates spinal stability by adding together six radio- 
graphic and clinical components, with a score ranging 
from 0 to 18 (Table 1). A clinical example is shown in 
Figure 1. The total score is divided in three categories 
of stability: stable (0-6 points), potentially unstable 
(7-12 points), and unstable (13-18 points). In addition, 
the SINS score can also be analyzed as a binary indicator 
of surgical referral status: stable' (0-6 points) or current 
or possible instability (7-18 points). Surgical consultation 
is recommended for those patients with a score of >7 
(Table 2). 

The SINS performance was previously tested among 
members of the SOSG. This study revealed a sensitivity 
and specificity for potentially unstable or unstable le- 
sions of 95.7% and 79.5%, respectively. It demonstrated 
near-perfect inter- and intra-observer reliability for dif- 
ferentiating the three clinical categories of stability [8]. 
Patients with spinal metastases receive the majority of 
their care from oncologists who often find evaluating 
stability difficult. Therefore, before the widespread use 
of SINS can be recommended and studied prospectively, 
its reliability and validity has to be established among 
oncologists. The purpose of this study is to assess the re- 
liability and validity of the SINS score among radiation 
oncologists. We envision that the SINS can be used in 
the future to assist in directing appropriate evaluation 
and referral of these complex patients to spine surgeons 
when necessary. 

Methods 

Patient cases 

A radiation oncologist and spinal surgeon not related 
to or participating in the study compiled 30 de-identified 
cases of metastatic spine disease. The cases were com- 
prised of 13 females and 17 males, with ages ranging 
from 40-79 and were reviewed at ten international sites. 
Twelve primary pathologies were represented with lung 



Table 1 The SINS classification according to Fisher et al. [4] 



Score 

Location 

Junctional (occiput-C2, C7-T2, Tl 1-Ll, L5-S1) 3 

Mobile spine (C3-C6, L2-L4) 2 

Semirigid (T3-T10) 1 

Rigid (S2-S5) 0 
Pain* 

Yes 3 

Occasional pain but not mechanical 1 

Pain-free lesion 0 
Bone lesion 

Lytic 2 

Mixed (lytic/blastic) 1 

elastic 0 
Radiographic spinal alignment 

Subluxation/translation present 4 

De novo deformity (kyphosis/scoliosis) 2 

Normal alignment 0 

Vertebral body collapse 

> 50% collapse 3 

< 50% collapse 2 

No collapse with > 50% body involved 1 

None of the above 0 

Posterolateral involvement of spinal elements^ 

Bilateral 3 

Unilateral 1 

None of the above 0 



*Pain improvement with recumbency and/or pain with movement/loading 
of spine. 

^Facet, pedicle, or costovertebral joint fracture or replacement with tumor. 

= 6), prostate = 6), breast {n = 5), and myeloma {n = 4) 
being most popular. Lesions were spread across the cer- 
vical (C0-C7) spine {n = 6), thoracic (Tl-TlO) spine 
{n = ll), thoracolumbar (T11-L2) junction (a? = 8), and 
low lumbar (L3-L5) regions {n = 5). Information provided 
included demographic data, histopathological diagnosis, 
and relevant clinical information including a pain de- 
scription that emphasized the relationship of the pain to 
activity and rest; the purpose being to differentiate onco- 
logic from mechanical or movement related pain. Plain 
radiographs. Computed Tomography (CT) images, and 
Magnetic Resonance (MR) images followed. CT scans 
were provided for each assessment. 

To determine a validity reference or surgical consensus 
standard, 11 fellowship trained oncology spine surgeons 
from both neurosurgical and orthopedic backgrounds, 
all experienced with the SINS classification, rated the 
cases once. None of the raters had been involved in the 
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Figure 1 A 67-year-old man with l<nown metastatic small cell lung cancer presents with interscapular back pain that is exacerbated by 
movement and relieved with rest. Computed Tomography (CT) (A: left parasagittal, B: midline parasagittal, C: right parasagittal, D: coronal, and 
E: axial) and Magnectic Resonance (MR) images (F: sagittal, T2-weighted) outline the key details of this T5 lesion. Total SINS score = Semirigid (T5) 
spine, 1 point; 'Mechanical' pain (yes), 3 points; Lytic lesion, 2 points; De novo kyphotic deformity without subluxation/translation, 2 points; >50% 
vertebral body collapse, 3 points; and bilateral posterolateral spinal element involvement, 3 points. Total SINS score =1 + 3 + 2 + 2 + 3 + 3=14 
(unstable, surgical consultation is recommended). 



care of any of the cases. The consensus standard score 
was the value that occurred most frequently. In cases 
where ties occurred, the two most frequent scores were 
averaged to calculate the final value. 

After undergoing an instructional session and case 
examples on the application of the SINS, 33 radiation 
oncologists from North America, Europe, and Asia inde- 
pendently scored the SINS on the same 30 cases. Six to 
eight weeks later, after the order of cases was changed to 
minimize recall bias, the cases were scored again. The 
radiation oncologists received a stipend for scoring the 
cases. None of the cases in the instructional session were 
used in the actual study cases. 



For each case, the SINS score was rated and analyzed 
as a total (0-18) point score, as three clinical categories 
(0-6 points indicating a stable spine, 7-12 points indicating 
potentially unstable spine, and 13-18 points indicating an 
unstable spine), and as a binary scale (stable' (0-6 points) 
or current or possible instability' (7-18 points)) (Table 2). 
This study received ethics committee approval. 

Statistical analysis 

Inter-observer agreement was calculated for both the 
radiation oncologists and the surgeons based on the 
first review using the Fleiss' kappa for multiple raters 
[9]. Confidence intervals for the kappa estimates were 



Table 2 SINS scores organized as a total score, three-clinical categories, and binary scale with their corresponding 
levels of stability where surgical consultation is recommended for a total score > 7 



Total Score (0-18 SINS) 1 


2 3 4 5 6 


7 8 9 10 11 12 13 14 15 16 17 18 


Three Clinical Categories (3-point) 


Stability 


Potentially Unstable Unstable 


Binary Scale (2-point) 


Stability 


Current or potential instability; 
Surgical consultation recommended 
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calculated by means of bootstrapping, simulating 1,000 
kappa estimates, and using the 2.5 and 97.5 percentiles 
for the confidence intervals. Intra-observer reproducibility 
was assessed for the radiation oncologists using the aver- 
age Cohens kappa [10]. The kappa coefficients (k) were 
calculated for the three clinical categories (3-point scale), 
and the binary scale (2-point scale) and interpreted ac- 
cording to the widely accepted Landis and Koch [11] grad- 
ing system (Table 3). The intraclass correlation coefficient 
(ICC) was used to measure both the inter- and intra- 
observer agreement for the total point score. As a measure 
of validity, the average kappa for agreement between the 
radiation oncologists' rating and the surgeons consensus 
standard was used. The magnitude of agreement guide- 
lines of the kappa, ranging from "no agreement" to 
"almost perfect agreement", was also used (Table 3). All 
statistical analyses were performed with Stata 12.0 for 
Windows (StataCorp LP, College Station, TX, USA). 

Role of the funding source 

AOSpine International provided resources necessary to 
conduct this study including stipends to the radiation 
oncologists who reviewed the cases. They facilitated 
strategic project meetings with the co-authors and assisted 
in the study management process. The authors were not 
compensated. 

Results 

Inter-observer agreement 

Among surgeons, substantial (k = 0.65) agreement for 
the three clinical categories and excellent (k = 0.83) 
agreement for the binary scale was achieved. The agree- 
ment among the radiation oncologists for the three clinical 
categories and binary scale was moderate (k = 0.54) and 
substantial (k = 0.76), respectively (Table 4). Inter-observer 
agreement of the total point score was ICC = 0.85 among 
the surgeons and ICC = 0.80 among radiation oncologists. 
Among radiation oncologists, analysis of the separate SINS 
components showed excellent level of agreement for loca- 
tion (k = 0.94) and pain (k = 0.88) and moderate agree- 
ment for bone lesion quality (k = 0.55), spinal alignment 
(k = 0.42), vertebral body collapse (k = 0.57), and postero- 
lateral involvement of the spinal elements (k = 0.43). 



Table 3 Levels of agreement for k statistic levels 


K Value 


Level of agreement 


0.00-0.20 


Slight 


0.21-0.40 


Fair 


0.41-0.60 


Moderate 


0.61-0.80 


Substantial 


> 0.80 


Excellent 



NOTE. Data adapted from Landis and Koch [11]. 



Intra-observer reproducibility 

Among radiation oncologists, intra-observer reproduci- 
bility was substantial (k = 0.65) for the three clinical cat- 
egories and excellent (k = 0.80) for the binary scale 
(Table 4). Intra-observer reproducibility of the total 
point score among radiation oncologists was ICC = 0.88. 
Analysis of the separate SINS components showed ex- 
cellent reproducibility for location (k = 0.96) and pain 
(k = 0.91), substantial consistency for bone lesion quality 
(k = 0.68), spinal alignment (k = 0.63), and vertebral body 
collapse (k = 0.63), and moderate agreement for postero- 
lateral involvement of the spinal elements (k = 0.58). 

Validity 

The level of agreement (average Cohen s kappa) between 
the spine surgeon s consensus standard and the radiation 
oncologists for the three clinical categories was 0.61 and 
0.85 for the binary scale. Table 5 outlines the three clin- 
ical categories (0-6 stable, 7-12 potentially unstable, and 
13-18 unstable) rated by the radiation oncologists cross- 
tabulated with the spine surgeons opinion, the gold 
standard. None of the unstable cases and only 18 of 561 
(3%) of the potentially unstable cases, as determined by 
spine surgeons consensus standard, was rated as stable 
by the radiation oncologists. These potentially unstable 
and unstable cases were rated a score (>7) for which sur- 
gical consultation is may be prudent. Fourteen percent 
of cases rated stable by the surgeons were considered 
potentially unstable by the oncologists. 

Discussion 

Metastatic spine disease requires a multidisciplinary ap- 
proach in order to provide optimal care; therefore, it is 
important to establish a clear framework and termin- 
ology when describing important parameters such as 
spinal stability to avoid variability in interpretation, re- 
ferral patterns and consequently patient management. 
SINS is a highly reliable, reproducible and provisionally 
valid assessment tool for radiation oncologists attempt- 
ing to answer a key question in tumor related spinal 
disease: Is the spine stable' or is there current or po- 
tential instability that warrants surgical assessment? 
Based on the current study, we propose that the SINS 
should become part of the routine clinical assessment 
of patients with symptomatic spinal metastases. Al- 
though SINS has not undergone prospective clinical 
evaluation, it has been derived through evidence based 
medicine methodology and is the closest thing to a gold 
standard we have. In development of a classification 
or tool such as SINS, it must be assessed for face, 
content and criterion validity, along with reliability. 
With these established SINS can undergo prospective 
validation and only then, if successful, will it be a true 
gold standard. 
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Table 4 Reliability analysis of the SINS among radiation oncologists and spine surgeons 



Profession 


Inter-observer agreement 




Intra-observer reproducibility 






Three-point scale (95% CI) 


Binary scale (95% CI) 


Three-point scale (95% CI) 


Binary scale (95% CI) 


Radiation oncologists 


0.54 (0.40-0.64) 


0.76 (0.56-0.88) 


0.65 (0.60-0.71) 


0.80 (0.74-0.86) 


Spine surgeons (Gold standard) 


0.65 (0.50-0.75) 


0.83 (0.65-0.93) 


n.a. 


n.a. 



The agreement for each individual component of the 
SINS score varied between moderate' and excellent'. In 
part, this may be attributable to the study design with 
limited information provided for each case compared to 
the true clinical situation. Trying to accurately mimic 
real practice is one of the inherent limitations of doing 
reliability studies and could serve to increase or decrease 
reliability. For example, more history would be obtain- 
able from real' patients with respect to movement- 
related pain that may strengthen agreement. Conversely, 
history taking varies and patients are sometimes vague 
and do not have a distinct pain character and this could 
increase variability. The authors are hopeful that SINS 
will increase awareness of movement related pain and 
lead to accurate capture of this important variable. In 
the study there was excellent reproducibility on the pain 
component (k = 0.91). Only moderate agreement was 
recorded for bone lesion quality, spinal alignment, verte- 
bral body collapse, and posterolateral involvement of the 
spinal elements. Similar findings were noted when the 
reliability of the SINS was tested among members of the 
SOSG [8]. We acknowledge that these components may 
have a greater degree of subjective interpretation but it 
may reflect the availability of appropriate imaging or the 
inability to scroll through reformatted images in the 
study setting. On the other hand, having representative 
images selected for the study could improve reliability. 
Similarly in real practice, scrolling through reams of 
images to obtain representative ones could lead to in- 
creased variability. Imperfect agreement on location may 
reflect a discrepancy in application of SINS or illustrate 
limitations in anatomical interpretation of the static im- 
ages provided. The only way these limitations will be 
truly assessed is during prospective evaluation in a 



Table 5 Cross tabulation of scores determined by the gold 
standard and categorization by radiation oncologists 





Gold standard 




Total 


Three-point Scale SINS by 
Radiation Oncologists 


Stable 


Potentially 
unstable 


Unstable 




Stable 


170 


18 


0 


188 


Potentially Unstable 


28 


409 


56 


493 


Unstable 


0 


134 


175 


309 


Total 


198 


561 


231 


990 



Gold Standard: score occurred most frequently among spine surgeons. 
Radiation oncologist first assessment (N = 30). 



clinical setting. While agreement between surgeons and 
radiation oncologists was favorable for the total (0-18) 
point score, this measure is far less clinically relevant 
than the binary scale. Finally, the radiation oncologists' 
lack of familiarity with some of the terms in SINS may 
also impact reliability. 

Many scoring systems aiming to select patients that 
may benefit from spinal surgery for metastatic disease 
exist. Almost all attempt to guide decision-making by 
predicting survival prognosis [12-17]. In addition to 
prognosis, other key factors require consideration, in- 
cluding spinal stability. Spinal metastases can cause 
spinal instability, which may present as severe pain, 
progressive deformity and/or neurological compromise, 
all potential indications for surgical intervention. Prior to 
the development of the SINS classification the assess- 
ment of spinal integrity lacked standardization and 
proved difficult especially for non-surgical members of 
the multidisciplinary care team. The goal of the SINS 
classification is to provide a framework that helps all 
healthcare professionals identif)^ patients with impending 
or existing spinal instability who may benefit from surgi- 
cal intervention. A SINS score of 7 or greater should 
prompt consultation with a spine surgeon, who can high- 
light the available surgical options during collabora- 
tive discussions on a patients optimal treatment. A 
potential limitation of this however is too many unneces- 
sary referrals, which is likely not economically respon- 
sible. Fourteen percent of cases rated stable by the 
surgeons were considered potentially unstable by the 
radiation oncologists. This must be assessed with SINS' 
prospective evaluation in a multicenter clinical setting. 

Reliability of the SINS has already been evaluated 
among members of the SOSG [8]. Before widespread 
clinical implementation the reliability and validity of the 
SINS should be investigated across healthcare specialties. 
A similar assessment of the SINS classification with 
radiologists has also been initiated. SINS may also 
prove valuable beyond the clinical setting as a research 
tool to better classif)^ patients and as instability is likely a 
strong confounder for several outcomes in treatment 
evaluation. 

Our study showed substantial inter-observer and ex- 
cellent intra-observer reliability for the binary SINS scale 
(stable versus unstable/potentially unstable) among radi- 
ation oncologists. Thirty cases were reviewed twice by 
33 radiation oncologists at ten international sites, with 
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sufficient power (>94%) [18]. Generalizability was en- 
hanced by a multicenter, international group of oncolo- 
gists who reviewed cases that represented the established 
epidemiological profile of metastatic spine disease. SINS 
is designed to be reliable and valid throughout the mobile 
spine. 

With appropriate diagnostic test parameter testing 
completed, steps will be taken to implement SINS inte- 
gration into clinical practice so multicenter prospective 
evaluation can be carried out. With an evidence based 
medicine process to develop SINS, followed by reliability 
and validity studies involving the key specialists who will 
use it, adoption into clinical practice should be feasible. 
Furthermore, it would appear limited training is neces- 
sary based on the reliability and validity performance of 
SINS amongst radiation oncologists after a brief instruc- 
tion and case example tutorial. 

Conclusions 

The Spinal Instability Neoplastic Score (SINS) is a simple, 
reliable, and valid tool to determine metastatic spinal in- 
stability and facilitate appropriate surgical referral. This 
study has shown that among radiation oncologists, the 
SINS binary scale provides a reliable tool for rating tumor- 
related spinal instability. 
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